Camera systems using filters and exposure times to detect flickering illuminated objects

ABSTRACT

The technology relates to camera systems for vehicles having an autonomous driving mode. An example system includes a first camera mounted on a vehicle in order to capture images of the vehicle&#39;s environment. The first camera has a first exposure time and being without an ND filter. The system also includes a second camera mounted on the vehicle in order to capture images of the vehicle&#39;s environment and having an ND filter. The system also includes one or more processors configured to capture images using the first camera and the first exposure time, capture images using the second camera and the second exposure time, use the images captured using the second camera to identify illuminated objects, use the images captured using the first camera to identify the locations of objects, and use the identified illuminated objects and identified locations of objects to control the vehicle in an autonomous driving mode.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/613,546, filed Jun. 5, 2017, which claims the benefit of thefiling date of U.S. Provisional Patent Application No. 62/508,467 filedMay 19, 2017, the disclosures of which are hereby incorporated herein byreference.

BACKGROUND

Autonomous vehicles, such as vehicles that do not require a humandriver, can be used to aid in the transport of passengers or items fromone location to another. Such vehicles may operate in a fully autonomousdriving mode where passengers may provide some initial input, such as adestination, and the vehicle maneuvers itself to that destination. Thus,such vehicles may be largely dependent on systems that are capable ofdetermining the location of the autonomous vehicle at any given time, aswell as detecting and identifying objects external to the vehicle, suchas other vehicles, stop lights, pedestrians, etc.

While such sensors come in many different configurations, as an example,such sensors may include (“light detection and ranging”) LIDAR sensors,radar units, cameras, etc. In the camera example, in addition toconfiguration, the cameras have various features such as gain, exposuretime, etc. which must be set to particular values in order to obtainuseful images. Typically, the exposure time is determined by algorithmsbased on the ambient lighting conditions and brightness of the lights tobe detected. As such, these exposure times are often very short, forinstance, on the order of microseconds. However, in the case ofilluminated objects, while the human eye may see a solid continuouslight, in actuality many illuminated objects actually flicker dependingupon the frequency of the power grid (for instance, at 60 Hz) or whetherthe light (such as a light emitting diode (LED)) utilizes “pulse-widthmodulated light” (PWM). If these cameras were to sample something thathas a short light pulse, then the likelihood of imaging that light pulsewithin a timespan of a few microseconds is low.

BRIEF SUMMARY

Aspects of the disclosure provide a system. The system includes a firstcamera mounted on a vehicle in order to capture images of the vehicle'senvironment, the first camera having a first exposure time and beingwithout an ND filter; a second camera mounted on the vehicle in order tocapture images of the vehicle's environment, the second camera having asecond exposure time that is greater than or equal to the first exposuretime and having an ND filter; and one or more processors. The one ormore processors are configured to capture images using the first cameraand the first exposure time; capture images using the second camera andthe second exposure time; use the images captured using the secondcamera to identify illuminated objects; use the images captured usingthe first camera to identify the locations of objects; and use theidentified illuminated objects and identified locations of objects tocontrol the vehicle in an autonomous driving mode.

In one example, the first camera and the second camera each include anear infrared filter. In another example, the second exposure time is onthe order of milliseconds. In this example, the second exposure time isat least 5 milliseconds and the first exposure time is no greater than 5milliseconds. In another example, the ND filter is selected according tothe second exposure time. In another example, the ND filter isimplemented at a pixel level for the second camera. In another example,the system also includes the vehicle. In another example, the one ormore processors are configured to use the images of the second camera toidentify illuminated images by identifying light from a PWM lightsource. In another example, the one or more processors are configured touse the images of the second camera to identify illuminated images byidentifying text generated by a plurality of PWM light sourcescomprising LEDs. In this example, the one or more processors are furtherconfigured to select the second exposure time based on a frequency ofthe PWM light sources. In another example, the one or more processorsare configured to use the images of the second camera to identifyilluminated images by identifying light from a light source whichflickers at a rate defined by a power grid that supplies power to thelight source. In this example, the one or more processors are furtherconfigured to select the second exposure time based on a rate defined bythe power grid. In another example, the second exposure time is a fixedexposure time. In another example, the first exposure time is a variableexposure time that is adjusted according to ambient lighting conditions.In this example, the second exposure time is always greater than thefirst exposure time. In addition or alternatively, the second exposuretime is a variable exposure time.

Another aspect of the disclosure provides a camera for use on a vehicle.The camera includes a set of photodiodes, an ND filter arranged tofilter light before the light reaches the set of photodiodes, and acontroller configured to expose the set of photodiodes using a fixedexposure time of at least 5 milliseconds in order to capture an image,wherein the exposure time allows the camera to capture light from a PWMlight source during the exposure time, the PWM light being located in anenvironment of the vehicle. In one example, the camera also includes anear-infrared filter arranged to filter light before the light reachesthe set of photodiodes.

A further aspect of the disclosure provides camera for use on a vehicle.The camera includes a set of photodiodes; an ND filter arranged tofilter light before the light reaches the set of photodiodes; and acontroller configured to expose the set of photodiodes using a fixedexposure time of at least 5 milliseconds in order to capture an image,wherein the exposure time allows the camera to capture light from alight source which flickers at a rate defined by a power grid thatsupplies power to the light source, the light source being located in anenvironment of the vehicle. In one example, the camera also includes anear-infrared filter arranged to filter light before the light reachesthe set of photodiodes.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional diagram of an example vehicle in accordance withaspects of the disclosure according to aspects of the disclosure.

FIG. 2 is an example external view of the example vehicle of FIG. 1 inaccordance with aspects of the disclosure.

FIG. 3A is an example functional diagram of a first camera in accordancewith aspects of the disclosure.

FIG. 3B is an example functional diagram of a second camera inaccordance with aspects of the disclosure.

FIG. 4 is an example chart of representative exposure times and lightpulses in accordance with aspects of the disclosure.

FIG. 5 is another example chart of representative exposure times andlight pulses in accordance with aspects of the disclosure.

FIG. 6A is an example series of images captured by the first camera inaccordance with aspects of the disclosure.

FIG. 6B is an example of images of a sign captured by the first cameraand an image of the sign captured by the second camera in accordancewith aspects of the disclosure.

FIGS. 7A and 7B are example images of traffic signal light in accordancewith aspects of the disclosure.

FIG. 8 is a flow diagram in accordance with aspects of the disclosure.

DETAILED DESCRIPTION

The technology relates to controlling a vehicle, for instance in anautonomous driving mode, based on information detected in the vehicle'ssurroundings. As an example, such information may be detected using oneor more cameras mounted on the vehicle. As noted above, generally suchcameras use very short exposure times when the light in the scene isstrong, so that the scene is not over-exposed (some or all colorssaturate, distorting color, or parts of the image just being white) andalso rapidly adjust the camera configuration in order to best captureambient lighting conditions. However, in the case of illuminatedobjects, while the human eye may see a solid continuous light, inactuality many illuminated objects actually flicker depending upon thefrequency of the power grid or whether the light utilizes PWM. If thesecameras were to sample something that has a short light pulse, or rathera pulse having both a brief period of being on with a longer period ofbeing off during a very short amount of time as discussed below, thenthe likelihood of imaging that light pulse within a timespan of a fewmicroseconds is low.

To address this problem, the exposure time of one or more of the camerasmay be adjusted to a period which would be sufficient enough to coverthe period of both the power grid as well as PWM lights, such as thoseused for brake lights, turn signals, reverse lights, some headlights anddaytime running lights, as well as LED informational road signs (e.g.construction information signs, variable speed limit signs, variabledirection-of-traffic signs, etc.).

As an example, brake lights are especially likely to PWM because theyoften have two brightness settings, one for tail lights (on for longperiods) then a more intense one for brake lights (on while the vehicleis braking). The change in intensity may be implemented by usingdifferent PWM duty cycles. In addition the camera may be fitted with afilter that drastically reduces the amount of light reaching the lenssuch as a neutral density (ND) optical filter or other darkening filterthat cuts light significantly, such as those that adds color tint andthus are not necessarily “neutral”. Thus, any of the examples belowusing an ND filter may be replaced with such darkening filters.

The filter used may be selected in order to reach a balanced amount oflight for a particular exposure time. Such filters may help to make thistimespan much longer and thus the likelihood of imaging theaforementioned short pulses increases dramatically.

This combination of features allows the one or more cameras with the NDfilters and longer exposure times to capture images which are closer towhat a human eye would see. As such, the images captured by the camerawith the ND filter and the exposure time may be more reliable foridentifying traffic signal lights, brake lights, or turn signals thatflicker at speeds which are indiscernible to the human eye, than imagescaptured with other cameras without these features. This in turn, wouldmake identifying illuminated objects a much simpler task and avoidsituations where flickering lights would be incorrectly identified asnot being illuminated. Moreover, some of the images may be taken by twodifferent cameras at the same time, making aligning the images andmatching objects between them, etc. significantly simpler to do. Ofcourse, this information may then be used to control the vehicle.

Example Systems

As shown in FIG. 1, a vehicle 100 in accordance with one aspect of thedisclosure includes various components. While certain aspects of thedisclosure are particularly useful in connection with specific types ofvehicles, the vehicle may be any type of vehicle including, but notlimited to, cars, trucks, motorcycles, busses, recreational vehicles,etc. The vehicle may have one or more computing devices, such ascomputing devices 110 containing one or more processors 120, memory 130and other components typically present in general purpose computingdevices.

The memory 130 stores information accessible by the one or moreprocessors 120, including instructions 132 and data 134 that may beexecuted or otherwise used by the processor 120. The memory 130 may beof any type capable of storing information accessible by the processor,including a computing device-readable medium, or other medium thatstores data that may be read with the aid of an electronic device, suchas a hard-drive, memory card, ROM, RAM, DVD or other optical disks, aswell as other write-capable and read-only memories. Systems and methodsmay include different combinations of the foregoing, whereby differentportions of the instructions and data are stored on different types ofmedia.

The instructions 132 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions” and “programs” may be used interchangeably herein.The instructions may be stored in object code format for directprocessing by the processor, or in any other computing device languageincluding scripts or collections of independent source code modules thatare interpreted on demand or compiled in advance. Functions, methods androutines of the instructions are explained in more detail below.

The data 134 may be retrieved, stored or modified by processor 120 inaccordance with the instructions 132. For instance, although the claimedsubject matter is not limited by any particular data structure, the datamay be stored in computing device registers, in a relational database asa table having a plurality of different fields and records, XMLdocuments or flat files. The data may also be formatted in any computingdevice-readable format.

The one or more processor 120 may be any conventional processors, suchas commercially available CPUs. Alternatively, the one or moreprocessors may be a dedicated device such as an ASIC or otherhardware-based processor. Although FIG. 1 functionally illustrates theprocessor, memory, and other elements of computing devices 110 as beingwithin the same block, it will be understood by those of ordinary skillin the art that the processor, computing device, or memory may actuallyinclude multiple processors, computing devices, or memories that may ormay not be stored within the same physical housing. For example, memorymay be a hard drive or other storage media located in a housingdifferent from that of computing devices 110. Accordingly, references toa processor or computing device will be understood to include referencesto a collection of processors or computing devices or memories that mayor may not operate in parallel.

Computing devices 110 may include all of the components normally used inconnection with a computing device such as the processor and memorydescribed above as well as a user input 150 (e.g., a mouse, keyboard,touch screen and/or microphone) and various electronic displays (e.g., amonitor having a screen or any other electrical device that is operableto display information). In this example, the vehicle includes aninternal electronic display 152 as well as one or more speakers 154 toprovide information or audio visual experiences. In this regard,internal electronic display 152 may be located within a cabin of vehicle100 and may be used by computing devices 110 to provide information topassengers within the vehicle 100.

Computing devices 110 may also include one or more wireless networkconnections 156 to facilitate communication with other computingdevices, such as the client computing devices and server computingdevices described in detail below. The wireless network connections mayinclude short range communication protocols such as Bluetooth, Bluetoothlow energy (LE), cellular connections, as well as various configurationsand protocols including the Internet, World Wide Web, intranets, virtualprivate networks, wide area networks, local networks, private networksusing communication protocols proprietary to one or more companies,Ethernet, WiFi and HTTP, and various combinations of the foregoing.

In one example, computing devices 110 may be an autonomous drivingcomputing system incorporated into vehicle 100. The autonomous drivingcomputing system may capable of communicating with various components ofthe vehicle in order to maneuver vehicle 100 in a fully autonomousdriving mode and/or semi-autonomous driving mode. For example, returningto FIG. 1, computing devices 110 may be in communication with varioussystems of vehicle 100, such as deceleration system 160, accelerationsystem 162, steering system 164, signaling system 166, navigation system168, positioning system 170, perception system 172, and power system 174(for instance, a gasoline or diesel powered motor or electric engine) inorder to control the movement, speed, etc. of vehicle 100 in accordancewith the instructions 132 of memory 130. Again, although these systemsare shown as external to computing devices 110, in actuality, thesesystems may also be incorporated into computing devices 110, again as anautonomous driving computing system for controlling vehicle 100.

As an example, computing devices 110 may interact with decelerationsystem 160 and acceleration system 162 in order to control the speed ofthe vehicle. Similarly, steering system 164 may be used by computingdevices 110 in order to control the direction of vehicle 100. Forexample, if vehicle 100 is configured for use on a road, such as a caror truck, the steering system may include components to control theangle of wheels to turn the vehicle. Signaling system 166 may be used bycomputing devices 110 in order to signal the vehicle's intent to otherdrivers or vehicles, for example, by lighting turn signals or brakelights when needed.

Navigation system 168 may be used by computing devices 110 in order todetermine and follow a route to a location. In this regard, thenavigation system 168 and/or data 134 may store detailed mapinformation, e.g., highly detailed maps identifying the shape andelevation of roadways, lane lines, intersections, crosswalks, speedlimits, traffic signals, buildings, signs, real time trafficinformation, vegetation, or other such objects and information. In otherwords, this detailed map information may define the geometry ofvehicle's expected environment including roadways as well as speedrestrictions (legal speed limits) for those roadways. In addition, thismap information may include information regarding traffic controls, suchas traffic signal lights, stop signs, yield signs, etc., which, inconjunction with real time information received from the perceptionsystem 172, can be used by the computing devices 110 to determine whichdirections of traffic have the right of way at a given location.

The perception system 172 also includes one or more components fordetecting objects external to the vehicle such as other vehicles,obstacles in the roadway, traffic signals, signs, trees, etc. Forexample, the perception system 172 may include one or more LIDARsensors, sonar devices, microphones, radar units, cameras and/or anyother detection devices that record data which may be processed bycomputing devices 110. The sensors of the perception system may detectobjects and their characteristics such as location, orientation, size,shape, type, direction and speed of movement, etc. The raw data from thesensors and/or the aforementioned characteristics can be quantified orarranged into a descriptive function or vector and sent for furtherprocessing to the computing devices 110. As discussed in further detailbelow, computing devices 110 may use the positioning system 170 todetermine the vehicle's location and perception system 172 to detect andrespond to objects when needed to reach the location safely.

FIG. 2 is an example external view of a vehicle 100. As indicated above,the perception system 172 may include one or more sensors, such as oneor more cameras, which may be mounted on the vehicle at variouslocations. In this example, camera 200 (which may represent multiplecameras) is mounted just behind a front windshield 204 of the vehicle.This placement allows the cameras to capture a significant portion ofthe environment of the front of the vehicle. In addition, housing 210located on the roof panel 212 of the vehicle 100 may house one or moreadditional cameras mounted within the housing. Cameras within thehousing may be oriented at different directions in order to captureimages in the front of the vehicle, rear of the vehicle, and/or “driver”and “passenger” sides of the vehicle.

FIGS. 3A and 3B are example functional diagrams of cameras 300 and 350of perception system 172. One or both of cameras 300, 350 may be locatedat any one of the positions of camera 200 or within housing 210. Asshown, camera 300 includes a controller 302 which may include one ormore processors, configured similarly to processors 120, which maycommunicate and control operation of a set of photodiodes 304. In thisconfiguration, light entering the camera passes through one or morefilters before reaching the photodiodes 304. In this example, a filter306 may be a near infrared filter in order to block or filterwavelengths at or close to infrared light. Other additional filters mayalso be used.

Operation of camera 300 may enable the perception system 172 to captureimages of the vehicle's surroundings as well as process and identify nonlight emitting objects as well as light emitting objects. As notedabove, in order to provide the perception system 172 with the mostuseful images of such objects, the exposure time of the camera 300 maybe selected to be very short. For instance, during typical daylighthours, the ambient lighting when capturing images using camera 300 isgenerally very bright, so the exposure time that is chosen is typicallyvery short or on the order of microseconds, for instance 1 to 50microseconds.

In some instances, the camera 300 may be used to capture both “light”exposure images and “dark” exposure images in order to allow theperception system 172 and/or the computing devices 110 to identify bothnon-emissive (using the light exposure image) and light emissive objectsdark exposure image. To do so, a first image is processed by thecontroller 302 to determine an exposure value for capturing the averageamount of light (within a predetermined range) in the environment, forinstance using a logarithmic control for shutter time and a linearcontrol for the gain value. This exposure value is then used to capturethe light exposure image. A fixed offset value may then be added (orused to multiply) to one or more camera settings such as shutter timeand gain in order to use the same camera to capture the dark exposureimage. This process may continue such that the camera is used to capturea series of light and dark exposure images as shown in the example ofFIG. 4 discussed further below. Accordingly, the exposure time of thecamera 300 is variable according to the ambient lighting conditions, andmay, for instance, range from as little as a few microseconds to fewmilliseconds, for instance, up to 10 milliseconds. This upper boundlimit may be useful in order to limit motion blur caused by the camerabeing used on a moving vehicle.

As noted above, if the camera 300 is attempting to sample an image of anobject that has a very short light pulse, then the likelihood of imagingthat light pulse within a timespan of a few microseconds is low. Forinstance, using the example of a traffic light or any light powered bythe power grid in the US, the power grid frequency would be 60 Hz (16.66ms) with two half-periods of 8.33 ms in which the traffic light is atits maximum in the middle of the cycle. In other words, there is amaximum light event every 8.33 milliseconds as shown in the example plotof FIG. 4. In addition, more than 85% of the light is produced during25% of the cycle (for example that part of the light pulse above dashedline 410). Thus, with an exposure on the order of a microsecond, thelikelihood of capturing an image of the traffic light at the maximum orduring some portion of the cycle which would provide enough light isvery low, for instance, approximately 25% of the time which would beinsufficient and inefficient for the purposes of safely controlling avehicle in an autonomous driving mode. As such, camera 350, discussedfurther below, may also be used to capture such pulsed illuminatedlights.

As with lights that flicker according to the power grid frequency, PWMlights can also be difficult to discern with short exposure times. Asnoted above, LEDs, which are PWM, are commonly used in brake lights,turn signals, reverse lights, some headlights and daytime runninglights, as well as LED informational road signs (e.g. constructioninformation signs, variable speed limit signs, variabledirection-of-traffic signs, etc.) as well as various other types oflights which may be encountered by the perception system. For instance,PWM lights also have very short light pulses, typically operating atfrequencies of about 100-110 Hz with on-fractions of approximately 10%(though such frequencies can vary widely from 80 to 400 Hz with dutycycles from less than 10% to 80%). As an example, if a brake light usesa frequency of 100 Hz with 10% on-fraction as shown in the example plotof FIG. 5, a pulse of 1 millisecond of light is emitted, followed by 9milliseconds of no light, then another 1 millisecond of light, 9milliseconds of no light, and so on. Thus, depending upon when camera300 captures an exposure, during 1 millisecond the light will appear onin the image, and in the next 9, there will be no light from the brakelight. Again, this makes determining the state of a brake light fromsingle or small set of images difficult and in some cases impossible. Assuch, camera 350 may also be used to capture such pulsed illuminatedlights.

This can be especially problematic where the camera 300 is capturing animage of a light up road sign which includes text formed from LEDs. Thetext in images captured by camera 300 would only be partially visibleduring the exposure, and making it extremely difficult to identify thetext from a single image or even by combining a series of imagescaptured over time. This is because many signs that use LEDs for texthave different portions of the text lit for a different parts of the PWMcycle. In other words, one subset of LEDs may be lit in a first 10% ofcycle, another subset in a second 10% of the cycle, and so on. As such,only 1/10 of the text is illuminated at any given time.

For example, FIG. 6A demonstrates how a short exposure time, on theorder of microseconds used for fairly bright daylight ambient lightingconditions, can cause lighted signs with text that use PWM lights to beincoherent and impossible to decipher by the vehicle's perception system172 and/or computing devices 110. In this example, a series of 12 imagescaptured by a camera configured similarly to camera 300 (with shortexposure time and no ND filter) depicts a traffic sign with text formedfrom illuminated PWM lights, here LEDs. As discussed in the exampleabove, the top 6 images are captured as light exposure images 602 inorder to identify non emissive objects, while the lower 6 images arecaptured as dark exposure images 604 in order to identify emissiveobjects. However, as can be seen, while it is clear that the sign isilluminated, the text is incoherent. As such, camera 350 may also beused to capture such pulsed illuminated lights.

As shown in FIG. 3B, camera 350 includes a controller 352, comparable tocontroller 302, that may communicate and control operation of a set ofphotodiodes 354. In this configuration, light entering the camera passesthrough one or more filters before reaching the photodiodes 304. In thisexample, a first filter 356 may be a near infrared filter in order toblock or filter wavelengths at or close to infrared light, and a secondfilter 358 may be an ND filter. The ND filter may be selected in orderto tune the exposure time of the camera to a particular time frame. Forinstance, in order to achieve a 10 millisecond exposure time for camera350, a ˜1% ND filter may be used. This allows the camera 350 toeffectively increase the exposure time from camera 300 approximately 100times or more or less while still providing useful images of thevehicle's environment. In this regard, the exposure time desired may beused to determine the type of ND filter.

Using an ND filter allows for a longer exposure time by filtering outadditional light. In other words, the exposure time of camera 350 may bemuch greater than camera 300 while still capturing useful images ofobjects. As an example, the exposure time can be on the order ofmilliseconds, such as for instance 1 to 20 milliseconds or timestherebetween, such as at least 5 or 10 milliseconds. FIGS. 7A and 7Bdemonstrate the same image of a pair of traffic signal lights 730, 732that are both illuminated in the color green using a longer exposuretime, for instance, on the order of milliseconds. Image 710 of FIG. 7Ais captured without an ND filter while image 720 of FIG. 7B is capturedwith an ND filter, for instance using a camera configured similarly tocamera 350. Each of the traffic signal lights 730, 732 is identified bya white circle for ease of understanding, although these circles are notpart of the images themselves. As can be seen, the images 710 and 720demonstrate how the use of the ND filter eliminates most of the otherinformation (or other light), allowing the viewer, and the vehicle'sperception system 172, to pick out the illuminated traffic lights morereadily. Although not visible from black and white images, the ND filteralso preserves the light's color. For example, in image 710, the trafficsignal lights appear white with green halo, while in image 720, thetraffic signal lights appear as green circles.

Returning to the example of the power grid traffic light, where there isa maximum light event every 8.33 milliseconds and where camera 350'sexposure time is 10 milliseconds, the camera 350, using a 10 millisecondexposure time, is likely to capture an illuminated traffic light verywell as shown in FIGS. 4, 7A and 7B. No matter where a 10 ms exposuretime begins (shown by the different 10 millisecond example exposuretimes), the camera 350 is able to capture a significant amount of thelight, and the ND filter removes other information (or additional light)not needed for identifying the traffic signal light. Similarly, in theexample of a PWM brake light, where the brake light pulses at 100 HZwith a 10% on-fraction, the camera 350, using a 10 millisecond exposuretime, is likely to capture an illuminated brake light as shown in FIG.5. Thus, the likelihood of imaging the aforementioned types ofilluminated lights increases significantly.

FIG. 6B demonstrates a comparison between images of a lighted sign withtext that use PWM lights captured without an ND filter and using a shortexposure time, on the order of microseconds, and an image of that samesign captured with an ND filter using a longer exposure time, on theorder of milliseconds. Each of the images 610-618 and 620 depict thesame traffic sign having text formed by illumined PWM lights, here LEDs.As with the example of FIG. 6A, images 610-618 of the traffic signcaptured under fairly bright daylight ambient lighting conditions withno ND filter and a short exposure time can cause such traffic signs tobe incoherent and impossible to decipher by the vehicle's perceptionsystem 172 and/or computing devices 110. Again, images 610-618 may havebeen captured by a camera configured similarly to camera 300 with ashorter exposure time, on the order of microseconds, and no ND filter.While it is clear that the lights of the sign are at least partiallyilluminated, the text is incoherent. Image 620 may have been captured bya camera configured similarly to camera 350 with an ND filter and alonger exposure time, again on the order to milliseconds. In the exampleof image 620, the text of the sign is clearly legible and therefore morelikely to be deciphered by the vehicle's perception system, for instanceusing OCR or other techniques discussed below.

The ND filter may be implemented as a variable ND filter. For instance,the ND filter may be electrically-controllable semi-transparent LCD.

As an alternative to camera 350's configuration with an ND filter, theaperture and/or lense of the camera 350 may be reduced. For instance,camera 300 may have an f/2 aperture and no ND filter (where f refers tofocal length). However, instead of camera 350 having an f/2 aperture andND filter, such as an ND64 filter, camera 350 may have an f/16 aperture.The f/16 aperture has an 8 times smaller diameter (or 16/2), which is 64times (or 8{circumflex over ( )}2) less area and thus allows for 64times less light transmission. Accordingly, this would act similarly tothe ND64 filter.

In addition or alternatively, an aperture stop may be used. The smalleraperture, lens, and/or aperture stop may provide a better depth of fieldso that far and near things are simultaneously in focus. The smallaperture camera may also be used with or without a darkening filter suchas an ND filter.

Although the filters described above are depicted as lenses which filterlight for all of the photodiodes at one, other filter configurations mayalso be used. For example, the filters may be implemented at the pixellevel. In this regard, different groups of photodiodes within the samecamera may be configured with different filter configurations, such thata single camera includes filters configurations which would allow forthe configurations of both cameras 300 and 350 at the same time.

Example Methods

In addition to the operations described above and illustrated in thefigures, various operations will now be described. It should beunderstood that the following operations do not have to be performed inthe precise order described below. Rather, various steps can be handledin a different order or simultaneously, and steps may also be added oromitted.

As the vehicle moves along, the sensors of the perception system 172 maysample the vehicle's environment. Referring to cameras 300 and 350, inorder to produce sensor data for processing by the computing devices110, controllers 302 and 352 may control the functioning of therespective cameras. In this regard, the controllers may cause each ofthe cameras to capture images of the vehicle's environment usingvariable or fixed exposure times.

In the case of camera 300, this exposure time may be determined based onthe ambient lighting conditions as discussed above. For instance, theexposure time of the first camera may be on the order of microseconds(such a 1, 45, 80, 100, 500, or just under 1 millisecond or timestherebetween) and up to as long as 10 milliseconds depending upon theambient lighting conditions as discussed above. Thus, the camera 300 maycapture images of the vehicle's environment sequentially, and in somecases by switching between dark and light exposures, using a variableexposure time tied to the ambient lighting conditions.

In the case of camera 350, this exposure time may be predetermined basedon the type of ND filter used. As in the example above, for a ˜1% NDfilter, the exposure time of the camera 350 may be set to 10milliseconds. Thus, the camera 350 may capture images of the vehicle'senvironment sequentially using a 10 millisecond exposure time. However,unlike the variable exposure time of camera 300, the exposure time ofcamera 350 may be a fixed value. In other examples, the exposure time ofcamera 350 may be varied, but still longer than the exposure time ofcamera 300, for instance, to increase the likelihood of capturing anon-flickering traffic light when there is sun glare nearby.

The images captured by the cameras may then be processed to identifyobjects in the vehicle's environment using various known techniques suchas, for example, Training a deep net classifier or using a differentmachine learning technique such as cascade classifiers or support vectormachines (SVM), matching a known pattern, extracting local features andmatching them against a model, segmenting the image to find an object,matching a certain color, blob detection, comparing gradient images orhistograms, etc. For instance, the images from camera 300 may be used toidentify the location of objects, such as road markings (for instance,lane lines or other markers), other road users (for instance, vehicles,bicyclists, pedestrians, etc.), signs (for instance, stop signs, speedsigns, construction signs, etc.), cones, construction barriers, foreignobjects on the roadway (for instance, debris, animals, etc.), and insome cases illuminated objects. Of course, exposure times on the orderof microseconds are too short to capture some illuminated objects, suchas those that flicker according to a power grid or PWM (such as forLEDs), and thus some objects may be missed in these images, such as inthe images 602 and 604 as well as images 610-618.

Again, to address this issue, images captured by the camera 350 may beprocessed to identify illuminated objects, and in particularly, thosethat flicker due to the frequency of the power grid or PWM as discussedabove. At the same time, because an illuminated state of such flickeringlights can be discerned from a single or very few images captured by acamera configured as camera 350 as demonstrated by image 620 of FIG. 6B,as compared to processing thousands if not tens of thousands of imagescaptured by a camera configured as camera 300, this can save quite a bitof processing power. Moreover, even when camera 300 is able to capturesuch flickering light when the exposure time is longer, such as close to10 milliseconds, because lighting conditions change and because theexposure time of camera 300 is variable, the likelihood of capturingsuch flickering lights consistently is very low. In addition, whenconsidering the computing devices 110 must make driving decisions inreal time, this makes the use of the ND filter and longer fixed (or insome examples, variable) exposure time for the camera 350 an incrediblyuseful tool that allows the computing devices 110 to “perceive” thevehicle's environment closer to how a person would.

The images captured by the camera themselves and/or informationidentified from those images may be provided to the computing devices110 in order to assist the computing devices in making driving decisionsfor the vehicle. For instance, the status of a traffic signal, forinstance solid illuminated or flashing (such as with a flashing yellowlight), may be readily determined from the images from camera 350 andused to control how the vehicle responds to the traffic light. Inanother example, the status of an LED brake light, LED headlamp, LEDlights of emergency vehicles, or LED turn signal light of anothervehicle may be readily determined from the images from camera 350 andused to control how the vehicle 100 interacts with that other vehicle,such as waiting for the other vehicle to make a turn, moving around theother vehicle (if passing is possible), etc. Similarly, the text from asign that utilizes LED lights to provide textual information, such asno-right-on-red signs, construction information boards, some speed limitsigns, etc. may be readily recognized, for instance, using opticalcharacter recognition techniques (OCR). In many circumstances, allowingthe computing device 110 to “understand” such text may make thecomputing devices more apt to respond to changing environments caused byconstruction, public safety notices, or other information provided onsuch LED lighted signs. Moreover, some of the images may be taken by twodifferent cameras at the same time, making aligning the images andmatching objects between them, etc. significantly simpler to do.

FIG. 8 is an example flow diagram 800 in accordance with some of theaspects described herein. The example flow diagram refers to a systemincluding first and second cameras, such as cameras 300 and 350,respectively. In this regard, the first camera may be mounted on avehicle, such as vehicle 100, in order to capture images of thevehicle's environment. The first camera has a first exposure time andbeing without an ND filter, where the first exposure is a variableexposure time that is adjusted according to ambient lighting conditions.The second camera may also be mounted on the vehicle in order to captureimages of the vehicle's environment. The second camera has a secondexposure time that is greater than or equal to the first exposure timeand also has an ND filter. The second exposure time is a fixed (or insome examples, a variable) exposure time. The system also includes oneor more processors, such as processors of controllers 302, 352 and ofcomputing devices 110, which may be configured to perform the operationsof flow diagram 800. For example, at block 810, the one or moreprocessors capture images using the first camera and the first exposuretime. At block 520, the one or more processors capture images using thesecond camera and the second exposure time. At block 830, the one ormore processors use the images captured using the second camera toidentify illuminated objects. At block 840, the one or more processorsuse the images captured using the first camera to identify the locationsof objects. At block 880, the one or more processors use the identifiedilluminated objects and identified locations of objects to control thevehicle in an autonomous driving mode.

Although the examples above relate to controlling vehicles having anautonomous driving mode, identifying illuminated objects as describedabove may also be useful for other driving systems. For example, theinformation may be provided for display to passengers within suchvehicles having autonomous driving modes to provide context about howthe vehicle's perception system is perceiving the vehicle's environment.In another example, the information may be used to provide notificationsor warnings to a driver of a vehicle which may be operating in a manualor semi-autonomous (less than fully autonomous driving mode), such as toprovide a warning that the driver is going to go through a red light oranother vehicle is braking or turning.

Unless otherwise stated, the foregoing alternative examples are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the embodiments should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples; rather, the examples are intended to illustrate onlyone of many possible embodiments. Further, the same reference numbers indifferent drawings can identify the same or similar elements.

The invention claimed is:
 1. A method comprising: capturing images usinga first camera and a first exposure time, the first camera being mountedin order to capture images of an environment, the first camera beingwithout an ND filter; capturing images using a second camera and asecond exposure time, the second camera being mounted in order tocapture images of the environment, the second exposure time beinggreater than or equal to the first exposure time and having an NDfilter; using the images captured using the second camera to identifyilluminated objects; using the images captured using the first camera toidentify the locations of objects; and using the identified illuminatedobjects and identified locations of objects to control a vehicle in anautonomous driving mode.
 2. The method of claim 1, wherein the firstcamera and the second camera each include a near infrared filter.
 3. Themethod of claim 1, wherein the second exposure time is on the order ofmilliseconds.
 4. The method of claim 3, wherein the second exposure timeis at least 5 milliseconds and the first exposure time is no greaterthan 5 milliseconds.
 5. The method of claim 1, wherein the ND filter isselected according to the second exposure time.
 6. The method of claim1, wherein the ND filter is implemented at a pixel level for the secondcamera.
 7. The method of claim 1, further comprising, using the imagesof the second camera to identify illuminated images by identifying lightfrom a PWM light source.
 8. The method of claim 1, further comprisingusing the images of the second camera to identify illuminated images byidentifying text generated by a plurality of PWM light sourcescomprising LEDs.
 9. The method of claim 8, further comprising selectingthe second exposure time based on a frequency of the PWM light sources.10. The method of claim 1, further comprising using the images of thesecond camera to identify illuminated images by identifying light from alight source which flickers at a rate defined by a power grid thatsupplies power to the light source.
 11. The method of claim 10, furthercomprising, selecting the second exposure time based on a rate definedby the power grid.
 12. The method of claim 1, wherein the secondexposure time is a fixed exposure time.
 13. The method of claim 1,wherein the first exposure time is a variable exposure time, and themethod further comprises adjusting the first exposure time according toambient lighting conditions.
 14. The method of claim 13, wherein thesecond exposure time is always greater than the first exposure time. 15.The method of claim 13, wherein the second exposure time is a variableexposure time.
 16. The method of claim 1 wherein the second cameraincludes a set of photodiodes and the ND filter is arranged to filterlight before the light reaches the set of photodiodes, and the methodfurther comprises, exposing the set of photodiodes using a fixedexposure time of at least 5 milliseconds in order to capture an image,wherein the exposure time allows the camera to capture light from a PWMlight source during the exposure time, the PWM light being located in anenvironment of the vehicle.
 17. The method of claim 16, furthercomprising using a near-infrared filter arranged to filter light beforethe light reaches the set of photodiodes.
 18. The method of claim 1,wherein the second camera includes a set of photodiodes, the ND filteris arranged to filter light before the light reaches the set ofphotodiodes, and the method further includes exposing the set ofphotodiodes using a fixed exposure time of at least 5 milliseconds inorder to capture an image, wherein the exposure time allows the camerato capture light from a light source which flickers at a rate defined bya power grid that supplies power to the light source, the light sourcebeing located in an environment of the vehicle.
 19. The method of claim18, further comprising using a near-infrared to filter light before thelight reaches the set of photodiodes.